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ABSTRACT 


The International Space Station (ISS) is a multinational orbiting space laboratory that is built in 
cooperation with 16 nations. The design and sustaining engineering expertise is spread 
worldwide. As the number of Partners with orbiting elements on the ISS grows, the challenge 
NASA is facing as the ISS integrator is to ensure that engineering expertise and data are 
accessible in a timely fashion to ensure ongoing operations and mission success. Integrating 
international engineering teams requires definition and agreement on common processes and 
responsibilities, joint training and the emergence of a unique engineering team culture. ISS 
engineers face daunting logistical and political challenges regarding data sharing requirements. 
To assure systematic information sharing and anomaly resolution of integrated anomalies, the 
ISS Partners are developing multi-lateral engineering interface procedures. Data sharing and 
individual responsibility are key aspects of this plan. This paper describes several examples of 
successful multilateral anomaly resolution. These successes were used to fonn the framework of 
the Partner to Partner engineering interface procedures, and this paper describes those currently 
documented multilateral engineering processes. Furthennore, it addresses the challenges 
experienced to date, and the forward work expected in establishing a successful working 
relationship with Partners as their hardware is launched. 
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INTRODUCTION 


The assembly and operation of the 
International Space Station (ISS) is a 
complicated task that requires the integrated 
effort of organizations from all over the 
world. The number of space agencies and 
engineering sites participating in the daily 
operations of the ISS increases as more 
hardware is added. Coordinating the work 
of these multiple sites will become more 
difficult and more important as the ISS 
grows. 

Anomaly resolution is a particularly vital 
part of ISS operations and is a key activity 
of the engineering organizations. The rapid 
resolution of anomalies and quick return to 
nonnal operations is critical to the ongoing 
success of the ISS program. Because the 
ISS is a complex system, an anomaly and 
the steps taken to resolve it can impact other 
subsystems, even if there are no direct 
interfaces between the systems. Interactions 
between subsystems can be unforeseen or 
unanticipated, so interdisciplinary 
investigations of anomalies must be 
thorough and coordinated. And because the 
safety of the crew or vehicle can be 
involved, the investigation process must be 
timely. 

Strong, well-defined procedures are a vital 
foundation for communications and 
interfaces. Additionally, human interaction 
is heavily dependent on a set of common 
perceptions and assumptions that are not 
necessarily or easily documented. Both of 
these are a challenge when working in a 
multi-cultural and multi-lingual 

environment. This paper first describes 
Partner to Partner interaction in the early 
years of the ISS program. Second, the 
responses to several actual anomalies are 


discussed to illustrate the gradual evolution 
and development of a common anomaly 
resolution approach. And then the new 
anomaly resolution philosophy that arose 
from these experiences is explained. In the 
last section of this paper, three areas of 
forward work are identified and discussed; 
the need to complete the interface 
procedures; the need to create a common 
engineering culture; the need to define data 
and information sharing scope and 
methodologies. 


ORIGINAL INTERFACE PROCESSES 

For the first part of the ISS program, 
anomaly resolution and engineering 
interface processes developed from a 
mixture of processes from previous 
programs and ad hoc procedures instigated 
on the spot. A single multi-lateral 
engineering process did not exist. 

The first on-orbit operational interfaces and 
consequently engineering interfaces were 
between Russia and U.S.A in 1998 after 
their modules were launched and mated. 
This was followed by the launch and 
operations of the Canadian contribution in 
2001 . 

The U.S. and Russia came to the ISS 
program without a long history of working 
together, although some lessons were 
learned through their joint Shuttle-MIR 
missions. However, both Partners had a 
long history of human space flight and well 
established procedures for its execution. 
Both Partners initially implemented their 
traditional human space flight methods in 
the ISS. 
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The initial engineering interactions can be 
characterized by difficult communication 
and inefficient interfaces. Several factors 
contributed to these initial communication 
challenges; the initial contacts and interfaces 
were primarily limited to the management 
level which did not adequately communicate 
down to their respective engineering teams; 
lack of jointly available communication 
tools; difference in methods and culture and 
the lack of familiarity of the assumptions 
and perceptions between teams. 

The initial environment, characterized by 
difficult communication and inefficient 
interfaces, resulted in very challenging 
integrated anomaly resolution. Direct 
communication between U.S. and Russian 
engineers was difficult to set up. There 
were no facilities or methods to easily share 
engineering data. No shared agreement 
existed, formal or otherwise, on how to 
jointly resolve anomalies. The eventual 
joint decisions and agreements often had to 
go to the highest management levels for 
approval, an inefficiency that prolonged the 
process. 

In the U.S./Russian situation, the problem 
that the ISS program faced was the need to 
establish interfaces at the lowest levels to 
share information and an agreement on a 
common process to resolve anomalies. 

Over the first several years of on-orbit 
operations, ISS issues and daily activities 
made both Partners realize that cooperation 
and communication at all levels was 
important to ensure smooth operation of the 
ISS. They also gained more trust and 
understanding of each other. The desire and 
culture for integrated anomaly resolution 
grew. U.S. and Russian engineering 
counterparts started meeting more frequently 


and exchanging greater amounts of 
information. In particular, the 

environmental discipline engineering teams 
were driven together by the need to manage 
the shared ISS cabin atmosphere. The two 
Partners also became more familiar with 
each other’s processes. A common 
understanding and culture was developing. 
These shared understandings and the 
realization of their importance led to the 
desire to establish and formalize formal 
multi-lateral engineering interfaces. 

The U.S. and Canada have a long history of 
working together on the Space Shuttle 
program and are very familiar with each 
other and in many ways the ISS program 
was a continuation of that relationship. 
There is a significant Canadian presence in 
the NASA mission control center, which 
means that many discussions and 
interactions can be face to face. The 
extended joint operational history and the 
collocation of personnel resulted in a very 
efficient engineering interface process 
between the U.S. and Canada and the 
emergence of a single joint process that is 
employed by both, but driven by NASA’s 
operational experience. 

Cooperation between U.S. and Canadian 
engineers can be so close that the distinction 
between U.S. and Canadian engineers might 
not be apparent to an outside observer. Both 
meet regularly and have access to common 
mission execution and anomaly databases. 
While this close U.S./Canadian relationship 
has the benefit of quick and efficient 
cooperation for anomaly resolution, the lack 
of formal process definition has meant that 
sometimes decisions and conclusions were 
not elevated to the proper management 
levels for review and concurrence. Therefore 
one issue that emerged from this successful 
cooperation was the need to formalize the 
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coordination and approval processes that 
respected the prerogatives of each Partner. 
This was eventually addressed in the 
development of fonnal multi-lateral 
engineering interfaces. 


ANOMALY RESOLUTION 
EXAMPLES 

Many ISS onboard anomalies requiring an 
integrated response have been successfully 
resolved. These provide excellent examples 
of the integrated engineering interactions 
that must occur to resolve anomalies. Some 
examples illustrate the unexpected 
interactions and unintended consequences 
that can occur between subsystems. Others 
demonstrate the importance of keeping other 
engineering organizations infonned of the 
progress of seemingly non-integrated 
anomaly resolution. An example of 
coordinated anomaly resolution leading to 
an efficient and correct result is also 
discussed. 


Example 1 — Failed Power Distribution 
Box 

A power distribution box in the US segment 
of ISS automatically turned off (‘tripped 
off) as a self-regulated response to a 
problem. This box provides power to a 
power converter in the Russian segment. 
The trip created an immediate cross-partner 
impact because power suddenly was no 
longer being provided to the Russian 
segment. Because of the history of this type 
of power distribution box, both American 
and Russian engineers initially believed that 
this trip was a reoccurrence of a previously 
experienced problem with this particular 
design. American engineers continued to 
investigate in order to confirm this 
hypothesis and Russian engineers undertook 
no further investigation in the Russian 
segment. However, the American 


investigation eventually determined that the 
trip was the result of an external overcurrent 
event and that the American power 
distribution box had responded nominally to 
an unsafe external current level. Russian 
engineers resumed their investigation and 
subsequently determined that an internal 
short in the Russian power converter was the 
cause of the overcurrent. 

In this case, the integrated impact of the 
anomaly was immediately apparent to all 
parties. Though both American and Russian 
engineers initially assumed an incorrect 
cause of the anomaly and ceased integrated 
anomaly resolution, the exoneration of the 
American hardware led to the resumption of 
communication and integrated anomaly 
resolution. 


Example 2 — High Humidity Levels in 
Plant Growth Payload 

During the operation of a plant growth 
payload in the US segment, humidity levels 
within the payload rose to levels that 
threatened the success of the experiment. 
Payload engineers launched an investigation 
that concluded that an airflow restriction 
caused by a noise muffler was the cause of 
the high humidity. The payload engineers 
asked the crew to remove the muffler and 
humidity levels began to decrease. Learning 
of the muffler removal through their regular 
review of payload anomaly reports, program 
safety engineers subsequently questioned the 
impact of the muffler removal on noise 
levels in the US segment. Program safety 
and systems engineers then started a second 
investigation of the situation. Acoustic 
measurements confirmed that onboard sound 
levels had increased and the crew confirmed 
that the levels, while tolerable, were 
annoying. Systems engineers were able to 
adjust the ISS thennal system settings in a 
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way that controlled the humidity levels in 
the payload and allowed the muffler to be 
reinstalled. 

This example illustrates the potential pitfall 
of uncoordinated anomaly resolution: 
unintended consequences caused by a lack 
of communication. The payload engineers 
did not foresee the effect of the first solution 
to the problem, an increase in noise levels 
due to the removal of the muffler. 
Furthermore, the payload and systems 
engineers lacked an effective and 
established method to cooperate and 
collaborate. 

Coordination and communication between 
all participants, even those not known to be 
affected, may result in solutions that provide 
greater risk reduction and increase anomaly 
resolution effectiveness. This particular 
anomaly led to review and improvement of 
the interface procedures between the US 
payloads and systems engineers. 

Example 3 — Russian Oxygen Generation 
System 

The Russian oxygen generation system 
started experiencing periodic shutdowns and 
the Russian team started troubleshooting. 
Because oxygen generation and 
consumption is a basic requirement for 
continued ISS operation, the U.S. 
engineering team closely monitored the 
anomaly resolution process. This system 
electrolyzes water to generate oxygen and 
generates hydrogen as a by-product. The 
hydrogen is normally vented overboard and 
the oxygen output into the ISS is monitored 
for hydrogen by a gas analyzer. As part of 
the troubleshooting and recovery, the gas 
analyzer was disabled. Knowing the 
importance of this gas analyzer and that it is 


an important hazard control, the U.S. and 
Russian safety teams worked together 
throughout the troubleshooting to analyze 
the configuration and assure that adequate 
hazard controls were maintained. Based on 
this coordination, additional monitoring was 
implemented to maintain the required hazard 
controls. 

This was an excellent example of integrated 
anomaly resolution. The Russian team 
completely handled the engineering 
troubleshooting of the problem, which was 
not an integrated issue. Control of the 
integrated hazard, inadvertent release of 
hydrogen into the ISS cabin, is an integrated 
issue and the U.S. and Russian teams 
cooperated to keep each other infonned and 
implement appropriate safety precautions. 


NEW PARTNER TO PARTNER 
INTERFACE PROCESSES 

With the launch of hardware from other 
Partners on the horizon, the ISS Program 
started a new effort to document and 
formalize engineering procedures. These 
procedures build on previous experience and 
incorporate lessons learned. 

The ISS experience with anomaly resolution 
and the gradual evolution and development 
of integrated anomaly resolution processes 
led to two key realizations. First, the 
engineers who designed and built a system 
will always understand the system best 
within the constraints of ongoing ISS 
operations. This implied that while other 
Partners’ engineers might be asked to 
understand or validate certain aspects of the 
system, responsibility for the operation, 
maintenance and anomaly resolution of 
those systems should be left with the 
design/development engineers. The second 
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realization was that in the complex systems 
environment of the ISS, integrated systems 
affects and impacts were not always 
apparent and a Partner could easily affect 
another Partner through their actions without 
being aware of it. This realization implied 
that a certain amount of integrated 
information about system operations and 
anomalies must always be shared between 
Partners even when the potential impact is 
not readily apparent. These two realizations 
became the guiding factors in the 
negotiation and development of the multi- 
lateral engineering interface procedures. 

Through multilateral negotiations, 
agreements are being made to define how 
the ISS Partners will communicate 
engineering issues. Each Partner is 
responsible for managing engineering 
support and analysis for their specific 
systems and for identifying, documenting, 
and resolving anomalies involving their 
hardware and software in accordance with 
their own processes. Each Partner is 
required to share a certain level of 
information to allow other Partners to 
determine the potential effects on their 
systems. And NASA, as the overall 
integrator, must be aware of the issues being 
worked, and is required to ensure that 
information from one Partner that affects 
other Partners is appropriately disseminated. 

Standing technical bilateral and multilateral 
forums have been created for discussing 
both on-going nominal operation of the ISS 
systems and resolution of anomalies. The 
forums exchange engineering data as 
necessary to ensure the safe and continued 
operation of ISS. These forums are under 
the direction of the each Partner’s subsystem 
managers and meet as needed. 


Integrated anomaly resolution presents a 
unique challenge to the ISS. Distance, 
experience, culture, and languages separate 
teams. Thus, extensive work is being done 
to ensure agreements are in place prior to the 
launch of European Space Agency (ESA) 
and Japan Aerospace Exploration Agency 
(JAXA) elements. These agreements build 
on the experience to date in ISS anomaly 
resolution. 

Special recognition is also given to the 
importance of real-time anomaly resolution 
in a human space flight program. When an 
anomaly occurs, it is of utmost importance 
that any actions necessary to place the ISS 
and its crew in a safe configuration are taken 
immediately. After that, the next goal is to 
implement corrective actions that allow the 
safe and continued operations of the ISS for 
an extended period of time. While each 
Partner is responsible for detennining the 
root cause of a failure in their hardware, the 
new processes recognize the importance of 
establishing a multilateral forum that will be 
used to discuss the impacts of the failure. 

To date, these procedures have been defined 
conceptually in a generic document. Most 
importantly, agreements have been made 
stating each Partner is responsible: 

For participating in the appropriate 
bilateral and multilateral engineering 
forums to discuss integrated 
anomalies. 

For investigating, documenting and 
closing anomalies involving their 
hardware and software in accordance 
with their own processes. 

FORWARD WORK 

In order to successfully integrate 
international engineering organizations for 
successful ISS operations, there are three 
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areas of work that must be completed. First, 
the engineering interface procedures must be 
completed, tested and refined. Second, the 
data and infonnation exchange agreements 
and methodologies must be established. 
Third, a common international ISS 
engineering team culture must emerge. 

Completion of Integrated Procedures 

The generic procedures for sustaining 
engineering have been under development 
since 2004 and a basic version has been 
published. While the multi-lateral 
engineering philosophy will be common to 
all of the ISS Partners, it is recognized that 
there will be differences in specific 
implementation for each Partner. These 
differences are captured in annexes to the 
multi-lateral engineering interface 
document. To date, the only annex that is 
under development is the European Space 
Agency (ESA) Automated Transfer Vehicle 
annex. Work is scheduled to begin shortly 
for annexes regarding the Japanese 
Aerospace Exploration Agency (JAXA) 
Japanese Experiment Module and the ESA 
Columbus module. 


Prior to the launch of ESA’s or JAXA’s 
modules, it is hoped that multilateral Partner 
engineering team simulations and training 
sessions will be held for operations specific 
to each module. The goal being to gain a 
better understanding of each Partners’ 
processes and to exercise the multi-lateral 
interface and anomaly resolution 
procedures. Early participation from ESA 
and JAXA engineering teams in operations 
preparation activities such as simulations 
will provide a large benefit in terms of 
training and building a common 
understanding. This will enable the ISS 
program to test and refine and further 
develop the interface procedures. 


Data and Information Sharing 

Data and information sharing on a day to 
day basis in the ISS program still has to be 
clearly defined. The exchange of 
information and technical data has two 
major hurdles at present. First, industrial 
policy and privacy issues must be respected 
since the hardware providers are primarily 
commercial companies. Second, US export 
control regulations add a significant 
overhead and restriction to data and 
information sharing. 

In an industrial policy driven world it is not 
sufficient to state that ‘required data will be 
shared’. Companies building and providing 
hardware have an interest in retaining their 
own intellectual property. Therefore, the 
interface between Partners must be 
cooperative to resolve anomalies at an 
integrated level but restricted to protect a 
companies intellectual property. Therefore, 
for example, root cause analysis data cannot 
be provided to the ISS Program level. 

Following the concepts agreed to via ISS 
Program Memorandum of Understandings at 
the government level and implementation at 
the National Space Agency levels, the scope 
of the Partner data sharing requirements are 
defined. These agreements must be 
respected in any interface procedure data 
sharing agreements. 

NASA has experienced difficulties in 
providing required data and personnel to 
international Partners due to the U.S. export 
control regulations. This has particularly 
hindered timely information exchange and 
has a potential to impact the integrity of the 
engineering interface procedures for 
anomaly resolution. Therefore, NASA must 
work to achieve a solution that follows U.S. 
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export control regulations while assuring 
ISS crew and vehicle safety. 

Assuming all data and information sharing 
agreements are resolved, the next issue to 
conquer is the means through which this 
data will be exchanged and a common data 
exchange format. Adaptation of existing 
tools will be required in order to easily 
exchange information between the Partners. 
NASA plans to expand its existing tools and 
provide additional access for Partners. ESA 
has already started development on an 
internal anomaly tracking system. 

Common Culture 

Developing a common culture for a truly 
integrated multi-lateral engineering team 
that is spread around the globe will be a 
challenge. Developing a common culture 
and a common attitude is an important part 
of team development. Experience has 
shown that an important factor in developing 
the culture of the U.S. engineering team is 
spending time together. By working and 
spending much time together, team members 
develop a common outlook. 

As seen from the initial joint operations 
between Russia and the U.S., a common 
culture will emerge once it is necessitated 
through common experience. Although in 
the case of US and Russian this common 
experience was only realized through on- 
orbit operations, there is a potential that 
early involvement in ESA and JAXA 
operations preparation activities will 
improve this timeline. 

A mutual respect and understanding of each 
Partners operations approach and 
methodology is required before a single 
culture will emerge. Additionally, all teams 


will have to recognize that the emergent 
engineering culture will be a unique 
engineering culture rather than the adoption 
of a single dominant culture. 


CONCLUSIONS 

The U.S., Canadian, and Russian Partners 
are cooperating well today. The ISS 
program is poised to integrate JAXA and 
ESA into the daily on-orbit operations of the 
program. Since initial joint ISS operations 
have started the multi-lateral engineering 
interfaces and anomaly resolution have 
advanced considerably. 

Initial international interfaces were based on 
improvised procedures and good intentions. 
Now procedures are being developed that 
should provide a robust, effective and 
efficient anomaly resolution process and 
engineering interface. Considerable gains in 
cooperation were realized simply through 
time spent working together. A key advance 
in this development is fonnalization of the 
philosophy that a certain level of data must 
be shared freely between Partners while 
maintaining respect for each Partner’s 
engineering expertise and prerogative. This 
is a true challenge in today’s political and 
commercial environment. 
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